1. Field of the Invention
This invention is related to the field of distributed file systems and, more particularly, to software used in recovering from failures in distributed file systems.
2. Description of the Related Art
There is an emerging trend in the Enterprise Storage industry for application servers to use file servers over a network. In these storage networks, applications access files on a file server over a network using a distributed file system protocol. Examples of such protocols are the Common Internet File System (CIFS) and the Network File System (NFS).
FIG. 1 illustrates an exemplary prior art storage environment with a single server. In this environment, client node 106 is coupled to a server node 102 via network 110. Client node 106 may be an application server. Network 110 may be, for example, an enterprise LAN or WAN. Application 108 on client node 106 may access files on storage 104 via server node 102. Server node 102 is coupled to storage 104 via storage network 100. Storage network 100 may be, for example, a Storage Area Network (SAN), although other network models may be used. In this example, server node 102 is a single point of failure. If server node 102 fails, application 108 cannot reach its files on storage 104. To overcome this, some enterprises may implement a clustered file server environment, where two or more servers are used to provide failover in the case of a file server failure. FIG. 2 illustrates an exemplary prior art storage environment with a cluster of three server nodes 102A, 102B, and 102C. If application 108 on client node 106 is accessing files on storage 104 via server node 102A, and server node 102A fails, application 108 on may failover to either server node 102B or 102C.
Some applications such as databases cannot tolerate I/O failures. In case of a file server failure, an application receives I/O failures for any file access. In case of a network connection failure, after the connection timeout the application receives a failure. If the file server fails to respond to any I/O requests (e.g. if there is a file server or network problem), the application may fail. To provide failover in the case of file server failure, the file server may be configured in a clustered environment. Even in a clustered environment, during the failover period of the file server, applications performing I/O operations may fail. In case of I/O failures, these applications may have to be re-instantiated and a recovery may need to be performed once the connection is established. This might be a costly and a time consuming operation. The separation of I/O and control paths made possible by storage networks has created a new manifestation of the problem.
Thus, it is desirable to provide a mechanism in storage networks for recovering failed I/O requests transparently, before the failed I/O requests reach the applications, to prevent the applications from failure, and to prevent any new I/O requests from failing. It is also desirable for this mechanism to be applicable in a variety of storage environments, including storage environments where applications on client nodes (e.g. application servers) issue direct I/O over a storage network such as a SAN to disks and control path I/O requests over a network to the file server.
The storage area network (SAN) model places storage on its own dedicated network, removing data storage from both the server-to-disk SCSI bus and the main user network. This dedicated network most commonly uses Fibre Channel technology, a versatile, high-speed transport. The SAN includes one or more hosts that provide a point of interface with LAN users, as well as (in the case of large SANs) one or more fabric switches, SAN hubs and other devices to accommodate a large number of storage devices. The hardware (e.g. fabric switches, hubs, bridges, routers, cables, etc.) that connects workstations and servers to storage devices in a SAN is referred to as a “fabric.” The SAN fabric may enable server-to-storage device connectivity through Fibre Channel switching technology to a wide range of servers and storage devices.